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Educational research and derelopaent (R&D) has often 
been characterized as a neat, linear sequence of discrete steps, 
V \ movtiiq iTom research through development to evaluation, an^d 

disseaination. Although the inadequacies of such linear models of 
/^educational research and developaent have been pointed out 
1 previously, these aodels have been so auch a part of R&D thinking 

that it is useful to point thea out again. In addition, the 
attributes of an interactive aodel are shown to surpass the linear 
aodel and to enhance the description of the R6D process, ihis aodel 
is discussed in the context of the Ljearning Research and Developaent 
Center (LRDC) where interactions take place aaong basic and applied 
research, developaent, evaluation, and disseainatlon. It is also 
discussed in conjunction with the theoretical requireaent of John 
. Deweyis theory of valuation which states that the value which peopl.e 
will attribute to a new product or procedure will be a function of 
how well they think it will satisfy soae recognized needs. The 
interplay of this interaction aodel and vaiuation theory is 
presented. (Author/BJG) - 
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EVALUATION IN A RESEARCH AND DEVELOPMENT CONTEXT 



William W. Cooley 

2 

Center for Advanced Study in the Behavioral Sciences 

Educational research and development has often been charac- 
tarized as a neat, linear sequence of discrete steps, moving from re- 
search through development to evaluation and dissemination (e.g. ,. 
Clark & Cuba, 1965). Although the inadequacies of such linear models 
of e.ducational research and development have been pointed out previbiis- 
ly Csea, for example, Baldridge & Johnson, 1972), these models have 
been so muck a part of R&D thinking that it is still useful to illustrate 
their shortcomings. Consider this scenario: The basic researcher. A, 
makes a psychological discovery regarding cognitive processes involved 
in comprehension by novice readers. B, the applied scientist, sees 
the possible implications of this discovery, and translates A's findings 
into validated operating principles with implications for instructional 
psychology. C, the developer, takes these new principles and builds 
a new reading program. D, the e valuator, field tests this product, finds 
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logical Research and Curriculum Development presented at the meeting 
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that it achieves C's objectives, and gives it a seal of approval. E, the 
publisher, develops the training and sales programs necessary for 
dissemination. Finally, teachers and kids, the consumers, live hap- 
pily ever after! Although this scenario illustrates the linear model for 
R&D, it certainly does not resemble what takes place^in educational 
research and development. 

An interactive model provides a better description of the R&D 
process. In an R&D environment, such as the Learning Research and 
Deveiopment Center (LRDC), interactions take place among basic and 
applied research, development, evaluation, and dissemination. Notice 
that the interactions are described as being among activities, not among 
distinct types of people. This is because individuals do not restrict thei 
attention to only one narrow aspect of R&D, but become involved in dif- 
ferent aspects of the R&D process. Much more could be said about the 
general nature of the interactions between research and development in 
a setting such as LRDC, but this paper deals with a more specific topic: 
evaluation and its interactions with psychological research and curricu- 
lum development. Examples of such interactions are drawn from the 
work being done at LRDC on a New Primary Grades Reading System 
(NRS)(Beck & Mitroff, 1972). 

Because '^evaluation'* has come to mean quite different things 
to different people, it is first necessary to clarify the mission of the 
set of activities called evaluation at LRDC. Evaluation involves the col- 
lection and organization of data into information relevant to judgments 
of value, the objects of these judgments being the products and proce- 
dures which LRDC is developing for use in the schools. Evaluation 
activities are called evaluative rege arch in recognition of the fact that 
the generation of such information or evidence, if it is to be believed by 



those for whom it is being prepared, must adhere to the same sound re- 
search principles which guide any disciplined inquiry. What distinguishes 
evaluative research from basic research is whether the propositions 
with which it deals are related to policy or to theory* Evaluative re- 
search investigates the validity of propositions which involve means and 
ends and their relationship to educational policy or practice, rather 
than propositions about cause and effect and their relationship to psycho- 
logical theory. - 

This is not to say that effective evaluative research is theory- 
free. Theory certainly is relevant, at least in the sense that orgauizing 
ideas are essential to guide the work. But in evaluation, the que^stidns ; 
to be investigated do not derive from theory. They derive from the 
world of action, and the results produced are fed back int > that world 
(see, for example, Colerran, 1972 for further discussion of this point). 
One theoretical requirement is a theory of valuation* A very ser^n.ceab^e 
theory was provided by John Dewey. A brief summary of this theory and 
its implications for the evaluation of educational systems is presented 
below. 

Quite simply, the theory says that the value which people will 
attribute, to a new product or procedure vill be a function of how well 
they think it will satisfy some recognized needs. LRDC*s mission is to 
improve education by providing schools with niore effective products 
and procedures than are currently available. To have such impact^ 
schools must implement what LRDC develops. They will do this only if 



A more complete summary can oe found in Dewey (1939) and 
in Evaluative Inquiry in Education, a forthcoming book by the author and 
Paul R. Lohnes. 



they see the need for it and are convinced that the product is demonstra- 
bly effective in fulfilling the need. The function of evaluative research 
is to provide information which demonstrates the need and shows how 
a particular innovation is a means to satisfy that need. Nothing star- 
tling ly new here, but if these value concerns are not kept in mind, the 
evaluator will tend to stop short of the total task, and, for example, 
show only that NRS does in fact teach the skills it was designed to teach. 
This latter demonstration is a necessary part of the evaluation task, but 
it is hot sufficient for determinations of, value. 

The specified need defines the end-in-view, and a means-ends 
continuum is proposed which establishes the relation between the inno- 
vative means and that end-in-view. The relative importance of the need 
and the credibility of the means-ends propositions, as seen by the val^ 
uer, will determine how highly the innovation will be valued by that \^l- 
uer. With respect to a new reading system, one might try to make the 
cape that it will increase grade equivalent scores on reading ability by 
six months for children from lower socioeconomic level families. If 
such an increase brings the mean for these children up to grade levels 
then the reading system will be valued by those who viewed grade -level 
achievement as a desired end. 

There may be more people, however, who are worried about 
the 21 million adults, products of our schools, who are unable to^read 
well e^nough to fill out a job application, vote, apply for a loan, obtain, 
information on such basics as medical aid, etc. If one could show the 
relationship between the use of NRS and the reduction of functional 
illiteracy in this country, then perhaps NRS would be valued even more 
highly. 



One might try to go even further and show how NRS could 
have been an effective weapon in the war on poverty, but such an ef- 
fort would have been in vain according to Jencks, Smith, A eland. Bane, 
Cohen, Gintis, Heyns, and Michelson (1972), because their evaluative, 
research suggests that the means-ends propositions which link the im- 
provement of schooling with the reduction of poverty are not valid. But 
the point is that the greater the established need and the more convincing 
the relationship between the^new means and the reduction of that need, 
the more highly the new means will be valued. 

Thus, one form of interaction among evaluation, research, 
and development is in the determination of need. Need i? not demon- 
strated by a small-scale market survey tafind out what consumers will 
buy next year, but rather by an analysis of critical educational and 
societal problems with the expectation that if this analysis guides de- 
yelopment work, what gets ''built" will more likely get used. Such ^ 
analyses would include a consideration of the major educational needs 
expressed by the critics of today^s schools, parent-community commit- 
tees on educational goals, and national educational policy groups, as 
well as consideration of the implications of empirical work of national, 
state, and local assessments, Harris polls (which publicized the illit- 
eracy problem mentioned above), etc. 

The other aspect of the valuing process is believing that the 
new means will move you toward the desired end-in-view. People have 
to believe something will work before they will buy it. This belief is a 
function of their seeing how it works as well as seeing that it works. 
Although buyers don'tinsist on understanding how a color TV set works 
before they buy it*, they are more willing to believe consumer satisfac 
tion data if it is accompanied by some plausible functional difference 



between one particular set and other sets on the market. Therefore, 
another aspect of the evaluation of NRS is making explicit its main 
features and the learning principles upon which it is based, and how 
they differ from alternatives. This, of course, is not necessarily done 
by someone with the name ''evaluator'/ on the door, but is another ex- 
ample of the necessary interaction among evaluation, research, and 
development activities. 

It is also important to recognize that valuation theory makes 
it clear that, given a means-ends continuum, value can be attached to 
both means and ends. Also, everyone **kno\ys'* that the ends dp not ^ 
justify the means. Thus, evaluation cannot restrict i't^ attention to out- _ 
comes. There is no sharp distinction between product and process, and 
both are subject to valuation. In fact, many*par,ents are more concerned 
about the kinds of experiences their children have in school than about 
the specifics of what their children are learning* Qne implication of\ 
this point for developers is the importance of making it clear to bthers 
what it is like for a child to be in their program. 

-1 The most demanding evaluation research task is generating 

information which clearly shows the learning effects of a new program, 
not just in the laboratory, cr in a carefully controlled developmental 
school, but also in the field, in schools representative of those to which 
the evaluation results are to be generalized. Field testing is compli- 
cated because of the variety of factors which affect the outconies of an 
educational system and because any program as intricate as NRS will 
surely get implemented in the field in a variety of ways. — - _ - 

The, variation in implementation which inevitably takes place 
from classroom to classroom provides an important focus for inter- 
actions among evaluation, research, and development. To discuss this 
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point, it is useful to refer to Pigure 1, which suggests that what a child 
learns during a given time interval (e.g. , during first grade) is a func- , 
tion of instruction, family,«.and peer group differences, and, most im- 
portantly, of the abilities with which the child began this time interval 
(T abilities). It has most certainly been established that each of the 
four s,ts of variables uniquely explains portions of the variance in 
abilities. Thus, all four must be considered in evaluation studies which 
attempt to identify instructional differences which affect how well children 
learn. 

This evaluation model does not imply pitting ?IRS classrooms 
against brand X classroonis and contrasting the two means. Very little 
is learned from such exercises. One ce^rtainly cannot assume that a 
given inethod will be uniformly implemented in the classroom. Also, 
any two reading programs will initially be alike in some respects and 
different in others. Reading programs in a sample of first-grade class- 
rooms (with or without NRS) might differ in the following ways: 

1. Degree of learner control (e. g. , how much of 
the program is prescribed, how much is explora- 

• - tory? ) 

2. Nature, firequency, and purpose of testing 
(e. g. , how often are diagnostic tests indi- 
vidually administered? ) ^ 

3. Individualization in goals (i.e.., is goal set- 

J 

ting done on an individual basis? ) 

4. Individualization in rate (i. e. , are children 
permitted to progress at different rates? ) 
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5. Teacher behaviors (e.g. , frequency and dis- 
tribution of contacts with individual children; 
percentage of contacts which are negative) 

6. Control of time (e.g., flexible or rigid; - - 
variable from day to day or from student 
to student depending upon need) 

7. Variety and quality of available instruc- 
tional materials (e. g. , are games and 
manipulables used in the classroom? ) 

8. Mastery learning (i. e. , is level of achieve- 
ment required before moving on to the next 
lesson? ) \ - ' -- ----- ----- \- -: 

9. Code-breaking approach (e. g. , degree to 
which grapheme /phoneme relationships 
are taught directly and practiced in isola- 
tion) _ 

Specifications for the measurement of instructional differences 
that are likely to make a difference in student achievement are critical- 
ly dependent upon successful interactions among specialists concerned 
with evaluation, research, and development, and the results have im- 
plications for all three. Examples from a recent dissertation by Leinhardt 
(1972) illustrate this point. '[.- \- 

The setting for her research was second-grade classrooms 
from LRDC's two developmental schools and from four school systems 
participating in the national Project Follow Through. Leinhardt developed 
measures of several kinds of implementation variables, and made obser- 
vations in 30 classrooms, all of which were using some form of LRDC^s 
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instructional program. She found that her measures of implementation 
were able to explain a considerable portion (46 percent) of the variance 
in abilities that was not explained by Tj^ abilities. An important 
point here is how different this finding is from the kinds of results which 
led'Jencks et al. (1972) to the conclusion that school differences aon't 
make a difference. They may be right, in part because a^bout two-thirds 
of the variance in output abilities is eliminated when school is used as 
the; unit of analysis, and in part because school variables (e. g. , cost/, 
pupil) only indirectly affect instruction. School differences may not 
make much of a diffeience, but classroom differences may be very 
important! : : 

Looking beyond the variance explained by the instructional 
variables, to their structure as predictors, it is possible to see the im- 
plications of this approach to evaluation for instruction theorists, de- 
velopers, evaluators, and consumers. For example, the Leinhardt data 
revealed a surprising relationship between frequency of testing and 
amount of learning which had taken place during the year. It suggested 
that teachers who overrode the system and tested less frequently than 
LRDG recommended, showed higher ability gains for their classrooms 
than did the teachers who tested more frequently. If the evaluator can 
convince the developer that this is a valid. finding, Lhen modifications can 
be made in the instructional program in its next approximation. If the 
instructional theorist decides that th« finding is a valid one, it may have 
implications for his (her) instructional model, at least the part about 
the necessity of constant monitoring of student progress once placement 
of the student has been achieved. 

It is the consumer who will benefit the most if evaluations de- 
scribe instructional effects in terms of multidimensional contributions , 
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instead of gross contrasts of pre-package plans (with no substitutions 
please!). Of course, it will take some getting used to in consumer- 
land. Buyers tend not to be provided information which tells them that 
they could add a $5 gadget to their TV set and have a picture of the same 
quality as a new $500 set would provide. But that - -nd of result 

our .evaluation model tends to produce. 

The typical contrast of gross treatments (e.g., NRS vs. Distar) 
usually results in a no- significant-difference finding, from which very 
little is learned. One reason for the rio-difference finding may be that 
although both developers made important contributions, they affected 
different aspects of the reading problem, with the same net effect for 
the two reading systems. Another possibility is that one of the systems 
is really superior (i.e. , based upon sounder principles) but was unevenly 
implemented, thus losing its potential impact. But the most important 
reason for moving from an analysis of variance mode, which contrasts 
gross treatments, to a multidimensional model of instructiona.1 differ- 
ences, has to do with the mission of LRDC. 

The mission of this R&D center is to improve education in 
elementary schools through an improved understanding of the instruc- 
tional process. An evaluation model which can contribute to those 
understandings is far more appropriate than one designed to promote 
products which mysteriously produce magical effects. 
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